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Abstract 

The problem of parameterization is often central to the ef- 
fective deployment of nature-inspired algorithms. However, 
finding the optimal set of parameter values for a combina- 
tion of problem instance and solution method is highly chal- 
lenging, and few concrete guidelines exist on how and when 
such tuning may be performed. Previous work tends to ei- 
ther focus on a specific algorithm or use benchmark prob- 
lems, and both of these restrictions limit the applicability of 
any findings. Here, we examine a number of different algo- 
rithms, and study them in a "problem agnostic" fashion (i.e., 
one that is not tied to specific instances) by considering their 
performance on fitness landscapes with varying characteris- 
tics. Using this approach, we make a number of observations 
on which algorithms may (or may not) benefit from tuning, 
and in which specific circumstances. 



Introduction and Background 

There exist many algorithms that are inspired by nature, and 
each has associated with it a set of parameters . These de- 
fine specific features or details of an algorithm that may be 
altered in order to change the behaviour or performance of 
the method (for example, in evolutionary algorithms, param- 
eters may include mutation rate or crossover probability). 
The problem of finding the optimal settings for these pa- 
rameters (often referred to as "tuning") is well-established 



(|Loboetal. 2007 Nannenetal. 


2008 Akay and Karaboga 


|2009t Birattaril |2009| Eiben and 


Smit| 2011), but Httle in- 



depth work has been performed on quantifying the benefits 
of tuning for a range of algorithms. We address this in the 
current paper, by investigating the precise benefits (or other- 
wise) of tuning for a number of different algorithms. More- 
over, we do this in a way that is independent of any specific 
problem, by using an approach based on fitness landscape 
characteristics. 

2013| l characterised a 



Previous work (Crossley et al. 



number of nature-inspired algorithms according to their 
performance on fitness landscapes with different features. 
However, the authors used the default parameter settings for 
each algorithm, which fails to reflect the fact that, in prac- 



et al. 2003 [ [Adenso-Diaz and Laguna 2006 Koster and 
Beney 2007 [ Ridge and Kudenko 2010 1. Here, we extend 
this work by quantifying the relative merits of tuning for a 
range of algorithms in a wide variety of fitness landscape 
scenarios. We achieve this by assessing both their tuned and 



untuned behaviour, using the methods described in Crossley 
|etaL| ( [20T3l ). 

In order to tune our selected algorithms, we use the no- 
tion of racing, which was first introduced in the field of 



machine learning ( [Maron and Moore 



ically, we use the F-race algorithm (IBirattari et al.', '2002 



1993 19971. Specif- 
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tice, methods are usually tuned prior to serious use ( [Leung 



Yuan and GaUagherj |2004| |Smit and Eiben 2009, Birattari| 
et al. 2010[ l, which has been extensively used to find the 
best possible set of parameter values for a given problem in 
a limited time. 

The rest of the paper is organised as follows: we first de- 
scribe our approach in the Methodology section, before pre- 
senting our experimental findings in the Results section. We 
conclude with a discussion of the implications of our results, 
and suggest further work. 

Methodology 

Our methodology may be summarised as follows: (1) select 
a number of nature-inspired algorithms, and obtain consis- 
tent source code for their implementation; (2) for each al- 
gorithm, find the best parameter settings (i.e., tune) over a 
number of different problems; (3) compare the performance 
of tuned and untuned algorithms. 

Algorithm selection 

We compare a number of nature-inspired algorithms, all of 
which are commonly applied to continuous function opti- 
misation (we use the same set as in Crossley et al. ('2013[l). 
These may be classified ( |Brabazon and O'Neill,|2006) as ei- 
ther social, evolutionary or physical. The social algorithms 
we select are Bacterial Foraging Optimisation Algorithm 
(BFOA) ( ,Passino,2002) , Bees Algorithm (BA) (Pham etaL] 
2006[l, and P article Swarm Optimisation (PSO) (Kennedy 
and Eberhartj |1995| l. The evolutionary algorithms selected 
are Genetic Algorithms (GA) ( (Goldberg, ,1989^ and Evolu- 



tion Strategies (ES) ( Back and Schwefel 1993| l, and phys- 
ical algorithms are represented by Harmony Search (HS) 
( |Geem and Kim| [2001 | l. We also include stochastic hill 
climbing (SHC) as a "baseline" algorithm; in contrast to 
Crossley et al. ( 2013| ) we exclude random search, as it has no 



Table 1 : A summary of the ranges selected for the various 
characteristics in the landscape generation methodology. 



parameters to tune. As before, we heed the observation that 
"Ideally, competing algorithms would be coded by the same 
expert programmer and run on the same test problems on the 
same computer configuration" (Barr et al. 1995| l. With that 



in mind, we use only implementations provided by Brown 



lee ( 201 1 1. Space prevents a complete description of specific 
implementation details for each algorithm, but full imple- 
mentation details can be found in |Brownlee| ( |201 1) , which is 
freely available and contains complete source code. 

Tuning 

Our fundamental goal is to investigate the pre- and post- 
tuned performance of our selected algorithms on landscapes 
with different general features, and thus identify character- 
istics of landscapes for which tuning may yield significant 



differences in algorithm performance. As Morgan and Gal- 



lagher (2010 1 observe, "Different problem types have their 
own characteristics, however it is usually the case that com- 
plementary insights into algorithm behaviour result from 
conducting larger experimental studies using a variety of dif- 
ferent problem types" (our emphasis). Rather than using ar- 
bitrary benchmark instances of problems in order to perform 
tuning, we use a landscape-based approach, as utilised in 
[Crossley et"STl ( |2013 V As Morgan and Gallagher'('2010) ex- 
plain, this Max-Set of Gaussians (MSG) method (Gallagher| 



and Yuan 2006 1 is a "randomised landscape generator that 
specifies test problems as a weighted sum of Gaussian func- 
tions. By specifying the number of Gaussians and the mean 
and covariance parameters for each component, a variety of 
test landscape instances can be generated. The topological 
properties of the landscapes are intuitively related to (and 
vary smoothly with) the parameters of the generator" We 
now describe the characteristics under study: 

Ruggedness of a landscape is often linked to its difficulty 
(Jones and Forrest 1995| l, and factors affecting this include 
(1) the number of local optima ( [Horn and Goldberg] |1994) l, 
and (2) ratio of the fitness value of local optima to the global 
optimal value ( [Malan and Engelbrecht| |2009[ |Merz| |2000|l. 
Other significant factors concern (3) dimensionality (Hendt-' 
|rass,,2009) (that is, the number of variables in the objective 
function), (4) boundary constraints (that is, the limits im- 
posed on the value of a variable) ( jKukkonen and Lampinen' 
[2005 ) , and (5) smoothness of each Gaussian curve (effec- 



tively the gradient) used to generate the landscape (Beyer 
[and Schwefel| [2002) 1 - a smaller value indicates a smoother 
gradient. For each characteristic, we use the same ranges as 
in |Crossley et al.|p013| l, summarised in Table[l] 

To produce a test set of problems, we use the MSG land- 
scape generator. For every value of every characteristic (in 
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Ratio of local optima 


0.1 
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to global optimum 










Dimensionality 


1 


10 


1 
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Boundary constraints 


10 


100 


10 


30 


Smoothness 


10 


100 


10 


15 



the range specified in Table [T]l we generate a set of five 
landscapes, which makes up the initial problem set for each 
value. We then use the F-racing methodology (BirattariJ 
et al. 2002 1 to find optimised parameters for each algorithm, 
over every value of every landscape characteristic used. We 
ensure that termination criteria are standardised, in order to 
ensure reasonable comparisons, and therefore use the num- 
ber of objective function evaluations to determine when to 
terminate an algorithm's run. We established, through initial 
experiments, that all selected algorithms generally converge 
within 20,000 objective function evaluations, so we use that 
as the specific value. 



Comparison 

We run each algorithm 100 times on each landscape in the 
set of landscapes generated for each particular characteristic 
value (when investigating smoothness, for example, we gen- 
erate 1000 different landscapes (100 each for smoothness = 
10 . . . 100), and run each algorithm 100 times on each land- 
scape). This is done first for all algorithms with 'default' 
parameter configurations, and then again, this time using 
the parameter configurations obtained through the F-Racing 
process. We measure the performance of each algorithm in 
terms of the mean {p) and standard deviation (ct) of the ex- 
act average error obtained, over all values for a particular 
characteristic. That is, we investigate the robustness of each 
algorithm to changes in the values for each characteristic, 
rather than their absolute performance on specific problem 
instances. This allows us to identify specific landscape fea- 
tures where tuning may make a significant difference, some 
difference, or no difference at all, for a particular algorithm. 

Results 

We find that the effect of tuning is varied across algorithms, 
and that they fit into three categories: Algorithms which do 
not benefit from tuning (ES), algorithms which only benefit 
significantly from tuning when a landscape is 'difficult' for 
the algorithm using default parameters (BA, HS, PSO), and 
algorithms which always benefit from tuning (BFOA, GA, 
SHC). We summarise our results in Table |2j the full datasets 



are available onlin^ We now examine in detail the per- 
formance of each algorithm; for each graphical spider plot, 
the further a line is from the origin, the smaller the average 
error (that is, the "larger" an area, the larger the degree of 
robustness, which is considered "better"). 

Bacterial Foraging Optimisation Algorithm 

There exists little discussion on the role of different parame- 
ters in the BFOA. While some elements of the search pattern 
are clearly altered by various parameters, it is very difficult 
to estimate values for these. In the original description of the 
BFOA ( |Passino| |2002| l, the parameter values were assigned 
based on observation of actual bacterial colonies. While this 
may be true to the nature-inspired concept, it is not nec- 
essarily the best way to obtain optimal performance from 
the algorithm. The combination of parameters offered by 
BFOA gives a highly configurable search environment. Pa- 
rameters such as step size and population size directly affect 
the potential area the algorithm can explore in a given num- 
ber of objective function calculations. Attraction and repul- 
sion weights, and the "space" over which these attraction 
and repulsion effects spread, work to control local optima 
avoidance. Parameters controlling the number of chemotac- 
tic steps before a reproduction step, and the number of repro- 
duction steps before an elimination-dispersal event, control 
the balance of exploitation versus exploration. Given that 
the search behaviour of the algorithm is highly configurable, 
it is unsurprising that BFOA is heavily reliant on tuning. 

Results for BFOA are shown in Figure [T] Across all char- 
acteristics, tuning offers a significant improvement on the 
average error and standard deviation of the performance - 
in many cases, improving the ranking of the algorithm from 
the largest average error to one of the smallest, and cop- 
ing well with the changing characteristics. We see the most 
significant improvement where boundary constraint ranges 
change, a characteristic that is heavily reliant on parameters 
which control the range at which new solutions are gener- 
ated (in the case of BFOA, this is the step size). Improve- 
ments are also shown for dimensionality and smoothness co- 
efficient, increasing the performance of BFOA where there 
is little gradient information in a large fitness landscape. 
Smaller improvements are demonstrated by the increasing 
number of local optima and the increasing attractiveness of 
these local optima, but tuning still benefits the algorithm 
considerably. 

In terms of the configurations selected by F-Racing, 
there is little variation in parameter values as characteristics 
change. Across all characteristics, and all values for those 
characteristics, there are only eight different configurations 
selected by racing. This suggests that, while it is difficult to 
find a good configuration, once it has been found, it is likely 
to be good for all similar problems. Tuning is vital to the 




I I Untuned BFOA 

I I Tuned BFOA 
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Figure 1 : Summary of results for bacterial foraging opti- 
misation algorithm. 

performance of the BFOA, but it is possible that by explor- 
ing problems using a similar methodology to that demon- 
strated here, we may create a 'bank' of promising configu- 
rations. 

Bees Algorithm 

The BA is considered to be an algorithm on which param- 
eterisation has little effect ( |Pham et al.| |2006 '). We observe 
that the BA is one of the best untuned performers in this 
study, offering weight to this argument for relative parame- 
ter insensitivity. In terms of adjusting the BA to cope with an 
increasing number of local optima, there are several param- 
eters which have an effect. Parameters such as the number 
of sites under investigation, the number of bees attributed to 
those sites, and the differentiation between sites and 'elite' 
sites are all factors which affect the searching behaviour of 
the algorithm to allow for greater flexibility as the modality 
of the problem landscape increases. 

Results for BA are shown in Figure |2] Post-tuning, we 
find that the BA selects the same parameter configuration, 
regardless of the number of local optima present in the land- 
scape. We then see that tuning has no effect on the ability of 
the algorithm to cope with increasing numbers of local op- 
tima. As long as the number of sites under investigation is 
greater than the number of optima, the algorithm is capable 
of dealing with modality. Coupled with the abandonment of 
'unpromising' sites, this means that 'too many' sites are not 
detrimental to the exploration pattern of the algorithm. 

We see the same pattern when increasing the ratio of lo- 
cal optima to the global optimum. As long as the number 
of sites under investigation covers the modality of the land- 
scape, the BA is not hampered by increasing levels of attrac- 
tiveness, regardless of parameter settings. The patch size 



Table 2: Mean (fi) and standard deviation (a) of the exact average error of algorithm performance (both untuned (UT) and 
tuned (T)). Smaller values imply more robustness to changes in a specific characteristic. 
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Figure 2: Summary of results for bees algorithm. 



Figure 3; Summary of results for evolution strategies. 



parameter of the BA controls the distance from a site bees 
are allowed to explore. This is the parameter which affects 
the search behaviour of the algorithm as boundary constraint 
size increases. The BA allows for full coverage of any sized 
search space, using scout bees to investigate new random 
sites to give 'teleportation' across the landscape. As with 
the number of local optima, we find the F-Races for the BA 
select the same parameter set for most boundary constraint 
sizes. We find that, post-tuning, the performance of the 
BA actually decreases slightly, suggesting the algorithm can 
cope less well with changes in boundary constraint size. We 
believe that the configurations may have become over-fitted 
to the landscapes used for tuning, and, while performance 
on the landscapes used for racing may have increased, the 
ability to search generalised landscapes has decreased. 

Dimensionality provides the most significant result in 
terms of pre-tuning and post-tuning performance of the BA. 
We observe little change in performance at one to three di- 
mensions - the point where the untuned algorithm is akeady 
performing well. As dimensionality increases beyond this, 
the effect of tuning becomes increasingly beneficial. We 



suggest that there is no increase in performance in other 
characteristics because these landscapes are not challenging 
enough to the BA to require adjusting the parameters. For 
the ranges of landscape characteristics on which we have 
tested the BA, it is clear that tuning generally makes little 
difference to the performance, as suggested by its original 
developer 

Evolution Strategies 

ES has the smallest number of parameters of all the al- 
gorithms studied here (excepting the baseline algorithm, 
stochastic hill climbing). The two parameters this form of 
ES offers are (1) population size and (2) number of children. 
It is suggested (Cant-Paz 2001') that altering these parame- 
ters adjusts selection pressure (that is to say, the greediness 
of the algorithm changes). The parameter configurations 
obtained through F-Racing are varied, implying that there 
do exist some configurations that are more successful than 
others. A range of configurations are selected across each 
characteristic - both in terms of different values for the two 
parameters, and different selection pressures when the two 



parameters are combined. 

Results for ES are shown in Figure |3] It is perhaps sur- 
prising to observe that the resuhs of using the tuned parame- 
ters show little or no change in performance across all char- 
acteristics. There is a small decrease in average error as the 
number of local optima changes, but the standard deviation 
is similar for both untuned and tuned, suggesting that while 
the average error has decreased very slightly, the ability of 
the algorithm to cope with increasing numbers of local op- 
tima is unchanged. For all other characteristics post-tuning, 
there is little change in both average error and standard de- 
viation across characteristics values (that is to say, the algo- 
rithm is no more capable of dealing with changes in these 
characteristics). This is perhaps consistent with the defini- 
tion of the two parameters the algorithm offers - selection 
pressure can only affect the way in which ES explores lo- 
cal optima, and there is no control over the area that is ex- 
plored around each point of interest, or any way to encour- 
age the algorithm to rapidly explore an increasingly large 
search space. 

We use a simple variant of ES, here, and there exist many 
other versions of the ES algorithm that offer a greater range 



of parameters (such as CMA-ES ( [Hansen and Kern 2004 1). 
ES clearly yields its best performance with an "out-of-the- 
box" parameter configuration, which means that it is quick 
to implement. However, our results suggest that there is little 
that can be done to improve the performance of this particu- 
lar variant. 

Genetic Algorithm 

The performance of the GA increases post-tuning, coping 
significantly better with increasing numbers of local op- 
tima, increasing boundary constraint range and an increas- 
ing smoothness coefficient. Results for the GA are shown 
in Figure]?] The parameters of the GA are not as intuitively 
linked to the exploration pattern as many of the other algo- 
rithms in the study. This particular GA offers four config- 
urable parameters: (I) population size, (2) 'bits' per param- 
eter in the representation, (3) crossover rate and (4) muta- 
tion rate. In experiments with a fixed number of objective 
function calculations, population size affects the number of 
generations the algorithm evaluates before terminating. A 
larger number of bits in a bit string representation allows 
more 'precise' solutions to be generated at the expense of 
having a representation which is less affected by mutation. 
Similarly to BFOA, there are a few configurations which re- 
occur across different characteristics and different charac- 
teristic values. It is probable that once a 'good' configura- 
tion has been found for a GA, it is applicable to 'similar' 
landscapes, which is consistent with the suggestion Gold- 
]berg (1989 ) that GAs are robust problem solvers, exhibiting 
approximately the same performance across a wide range of 
problems. 

With increasing dimensionality, the GA initially shows 
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Figure 4: Summary of results for genetic algorithm. 

promising results in terms of tuned performance, with a 
marked performance increase up to four dimensions. The 
benefit from tuning rapidly declines, however, until the 
tuned performance is worse than that of the tuned version. 
There are two possible explanations for this: the first is that 
the restriction on the number of objective calculations did 
not allow the F-Race algorithm to gather any meaningful 
performance data from the configurations. The second ex- 
planation is that we did not test a wide enough range of con- 
figurations - although two of the four parameters have def- 
inite ranges (mutation and crossover rates are percentages, 
thus generation was bounded between zero and one), so this 
is unlikely. 

Harmony Search 

The four parameters of HS all control different aspects of 
the search strategy. Memory size dictates how many promis- 
ing solutions can be stored - effectively, how many potential 
sites of interest are retained by the algorithm. Considera- 
tion rate and adjustment rate control how new solutions are 
generated. The consideration rate is the percentage chance 
that a solution based on one in memory will be generated 
(conversely, 1 -consideration rate is the chance a random so- 
lution is generated instead). The adjustment rate is then the 
percentage chance that the randomly chosen solution from 
memory will be adjusted. If so, the fourth parameter, which 
controls the maximum range at which solutions can be ad- 
justed, is used. If the adjustment does not occur, the con- 
sidered solution potentially occupies an additional slot in 
the memory - thus increasing the chance that this solution 
may be chosen for consideration again. The interplay be- 
tween these parameters is crucial, and it is somewhat hard to 
see how consideration rate and adjustment rate can directly 
affect the search strategy - unlike memory size and range. 




Figure 5: Summary of results for harmony search. 

which are more obvious. 

The resuhs for HS are shown in Figure |5] HS, Hke the 
BA, offers some of the lowest 'out of the box' average er- 
ror rates in this study. For most characteristics, there is lit- 
tle room for a performance increase post-tuning. Boundary 
constraint range proves to be the second-most challenging 
characteristic to HS pre-tuning, but post-tuning shows im- 
proved performance. The range values in all the configura- 
tions selected by F-Racing are much smaller than those in 
the 'out of the box' values, and this contributes significantly 
to the performance improvement when boundary constraint 
ranges are increasing. The consideration rate also decreases 
almost linearly as size increases - effectively, more random 
solutions are used instead of relying on the 'memory' . These 
random solutions allow the solution pool to jump from one 
position in the search space to another, encouraging a wider 
search space, and explaining the significant improvement as 
boundary constraint range increases. Dimensionality also 
yields an improvement in the tuned parameter performance 
of HS, in terms of both average error and abiUty to cope, as 
it rises. High dimension problems (seven and above) have a 
much higher consideration rate than the successful configu- 
rations for lower dimensionality, suggesting that a focus on 
exploitation rather than exploration is beneficial to the HS 
when dimensionality is high. This is the opposite case of 
what happens with boundary constraint range, as discussed 
above. 

Particle Swarm Optimisation 

PSO in this form has four parameters; these control the pop- 
ulation size, the maximum velocity of a particle, the bias 
towards the particle best solution and the bias towards the 
global best solution. With these parameters, it is possible to 
control the coverage of a search space (the number of parti- 



Figure 6: Summary of results for particle swarm optimisa- 
tion. 

cles), enforce a large search area of a small search area for 
each particle (the maximum velocity), and, through manip- 
ulation of the local and global best solution bias, control the 
capability of the algorithm to converge on a single solution 
or explore several areas of interest (optima avoidance). 

Results for PSO are shown in Figure [6] The parame- 
ters used cover a broad range of search behaviours, and, as 
such, we would expect to see a large improvement in particle 
swarm performance post-tuning. This holds true for most of 
our characteristics. Results for the number of local optima, 
for example, show a reasonable decrease in average error as 
the number of local optima increases, yet the standard devi- 
ation demonstrates no change, indicating that the algorithm 
is no more capable of dealing with increasing numbers of 
local optima post-tuning. Performance of PSO greatly im- 
proves on dimensionality post-tuning, in terms of both av- 
erage error and ability to cope as it grows. The F-Race al- 
gorithm for PSO selects the same configuration for all val- 
ues of dimensionality (except for 2 dimensions), implying 
that there is no specific parameter that requires adjustment 
to cope with the increase in dimensionality, but selecting a 
configuration which provides good exploration allows PSO 
to perform well as the size of the search space increases ex- 
ponentially. 

This trend continues across all characteristics, with F- 
Races often selecting the same configurations, regardless 
of characteristic values. As with the other swarming algo- 
rithms, we suggest that once a good configuration has been 
found, it is able to deal with a wide range of problems of a 
similar nature, regardless of the specific characteristics. The 
configurations selected are all varied in their parameters, 
and it is unexpected to see that there is no pattern to maxi- 
mum velocity as boundary constraint range increases. This 



is possibly because maximum velocity is an upper bound, 
and there are particles with randomly generated velocities 
below the maximum, so this parameter is less significant 
than it may initially appear. It would perhaps be interesting 
to consider the effect of having a minimum velocity on the 
increase in boundary constraint range, although this would 
also severely hamper exploitation. 

Stochastic Hill-Climbing 

With only a single parameter - the range at which new so- 
lutions are generated - the SHC algorithm does not offer a 
large amount of customisation. This single parameter is di- 
rectly linked to the search pattern and nothing else, and as 
there are no other parameters there is no interplay between 
parameters to consider. Arguably, therefore, SHC should 
prove the easiest algorithm to tune. 

Results for SHC are shown in Figure |7] All character- 
istics, barring dimensionality, show an improvement post- 
tuning. As the neighbourhood size is the range at which new 
solutions are generated, it is unsurprising that tuning im- 
proves algorithm performance as boundary constraint ranges 
change. As the number of objective function calculations 
is limited, despite having a larger neighbourhood size, the 
ability of the algorithm to effectively explore larger environ- 
ments is still restricted, therefore the average error does not 
decrease by as much as may be expected, and the ability of 
the algorithm to deal with increasing search space sizes im- 
proves only slightly. 

SHC demonstrates a large increase in performance and a 
greater ability to cope with more optima (a reduced stan- 
dard deviation) post- tuning. The parameter configurations 
selected for the number of local optima, the ratio of local 
optima and the smoothness all have a neighbourhood size of 
around 50% of the search space size. We suggest that the 
performance improvement for all of these characteristics is 
actually derived from the algorithm having configured itself 
properly for the search space size used as a default for all 
other characteristics, rather than tuning itself to best perform 
on any specific characteristic. 

Conclusions and Future Work 

In this paper we have built on previous studies of the perfor- 
mance of nature-inspired algorithms on fitness landscapes 
with different characteristics. Earlier work explored 'out of 
the box' parameter configurations, and we futher develop 
this by using an automated parameter configuration method- 
ology. This allows us to study the effect of tuning on dif- 
ferent algorithms, contributing significantly to the debate on 
when and how it is beneficial to tune specific algorithms. 

We observe that algorithms broadly fall into three cat- 
egories: (1) algorithms which do not benefit from tuning 
(ES), (2) algorithms which benefit from tuning when the 
problem is "difficult" enough to require tuning (i.e. when 
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Figure 7: Summary of results for stochastic hill-climbing 
algorithm. 

untuned performance is poor) (BA, HS, PSO), and (3) algo- 
rithms which always benefit from tuning (BFOA, GA, SHC). 
Dimensionality often offers the most significant improve- 
ment post-tuning in algorithms, particularly those with pa- 
rameters that increase the breadth of search space (swarm- 
ing algorithms are significantly better here than evolutionary 
algorithms). 

The methodology presented here is easy to implement, is 
computationally inexpensive, and offers considerably more 
information on the performance of an algorithm than using 
a standard set of benchmark problems. We hope that it will 
offer a foundation for future work on the experimental com- 
parison of nature-inspired algorithms, as well as a useful set 
of heuristics for practitioners to use in order to decide when 
and how to tune their methods. 
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